The computer implemented method and system disclosed herein, in general, relates to assistive technology. More particularly, the computer implemented method and system disclosed herein relates to assisting a user to learn and/or communicate in a visual communication language in one or more of multiple modes, for example, a visual mode, a text mode, an audio mode, etc.
A language is a medium for communicating thoughts, expressions, etc., of an individual. An individual with communication disabilities, for example, hearing impairment, speech disabilities, blindness, etc., faces challenges while communicating with others. A visual communication language, for example, a sign language, acts as a bridge for enabling communication between hearing impaired and inarticulate individuals and other individuals. For example, sign language uses visually transmitted sign patterns, hand gestures, etc., to communicate the thoughts, expressions, etc., of the individual.
An individual versed in a visual mode of the visual communication language faces challenges while communicating with others who are not versed in the visual mode of the visual communication language. For example, an individual with hearing impairment and speech disabilities who is versed in a sign language faces challenges while communicating with a blind individual. In another example, an individual with limited or no exposure to a sign language faces challenges in understanding and conversing with an individual having hearing impairment or speech disabilities who is versed in the sign language. In another example, an individual with hearing impairment and speech disabilities who is well versed only in one sign language format faces challenges in communicating with an individual having hearing impairment and speech disabilities who is well versed in a different sign language format. Hence, there is a need for assisting individuals with hearing impairment and speech disabilities and other individuals to learn and/or communicate in different modes of a visual communication language.
Currently, most communication assist devices are configured to recognize single handed gestures or a limited set of dual handed gestures for assisting individuals to communicate in a visual communication language. Some visual communication language gestures are based on occlusion and disambiguation which require sophisticated electronic paraphernalia for recognition. These sophisticated electronic paraphernalia are expensive and cannot be generalized to work with user defined gestures. Communication assist devices that are available are almost always pre-trained to work for a specific set of gestures and typically cannot adapt to an individual user's requirements, without intervention of a programmer or a device designer. Furthermore, systems that are designed as communication-assist devices may not function as adaptive tutors that generate and adapt lessons and tests to improve an individual user's skills in a visual communication language.
Furthermore, just as all speakers do not have the same fluency, all signers do not have the same skill or proficiency level to communicate using gestures. Therefore, if frames with gestures are sampled at a high rate, that is, with a low delay factor for a beginner, then the same words may be transcribed repeatedly or inter-gesture transitions may be transcribed erroneously as words in a communication language. Similarly, if the frames are sampled at a low rate, that is, with a high delay factor for an expert, then several words may not be transcribed as the rate at which the expert gesticulates may be much faster than the processing. Therefore, there is a need for automatically adjusting the delay factor for each user to determine the rate of sampling frames for processing.
Hence, there is a long felt but unresolved need for a computer implemented method and system that assists a user to adaptively learn, translate, and communicate a visual communication language in one or more of multiple modes. Moreover, there is a need for a computer implemented method and system that recognizes visual communication language gestures with occlusion and disambiguation and user defined visual communication language gestures based on the user's delay factor, while assisting the user to adaptively learn, translate, and communicate the visual communication language.